Netspeak - Assisting Writers in Choosing Words

NETSPEAK is a Web service which helps writers in finding alternative expressions for what they want to say.1 It provides a large index of writing samples in the form of ngrams, n ≤ 5, along with an efficient means to retrieve them by the use of wildcard queries. When in doubt about a phrasing, a user can get additional evidence by retrieving samples that match a given context. The figure below shows the results for a query where a user is interested in the two most frequently written words between “looks” and “me”. The first two columns give an idea about the customariness of each result, and the user can select the one most appropriate for her sentence. To provide a rich choice of writing samples we index the Google n-gram corpus which was compiled from a large portion of the English Web and which consists of more than 3 billion n-grams along with their occurrence frequencies [2]. We have developed a space-optimal inverted index based on minimal perfect hashing. The hash function maps the vocabulary V of the corpus to the storage positions of postlists. A hash function is perfect if it does not produce hash collisions for the key set V , and it is minimal if the number of storage positions required does not exceed |V |. The hash function is constructed with the CHD algorithm which produces a space overhead of 2.07 × |V | bits [1]. Moreover, the index provides a top-k retrieval strategy to find the n-grams matching a query; details can be found in [3]. The table below shows selected performance data of our index. NETSPEAK is currently deployed on a cluster of 15 computers. In a load test the service was measured to process about 10 000 queries per second.